System and method for tool mapping

ABSTRACT

A server receives datasets from mobile devices. Each dataset identifies a task selected in an augmented reality application of a corresponding mobile device and an identification of a tool detected at the corresponding mobile device. The server identifies tools present and absent at a dedicated tool board and compares an identification of the tools present and absent at the dedicated tool board with the tools detected at the mobile devices and the tasks identified at the mobile devices to generate a tool inventory and a tool compliance. The server generates an augmented reality content dataset for each mobile device to identify at least one of a missing tool, an incorrect tool, and a valid tool based on the tool compliance.

BACKGROUND

The subject matter disclosed herein generally relates to the technicalfield of machines that are configured to perform image processing.Specifically, the present disclosure addresses systems and methods touse head-mounted devices to identify and locate physical objects oridentify the lack of physical objects or equipment in expected locationsfor inventory purposes.

Workers in a factory use a variety of tools to perform their specificjob. However, those tools can be misplaced or lost thereby reducing theefficiency of the worker. Furthermore, unskilled workers can use thewrong tool on a machine to perform their job, which can lead to amalfunction of the machine and require costly repairs. Keeping track ofan inventory of tools in a large factory is increasingly difficult asthe number of workers and tools increases. Furthermore, a manualinventory process requires worker time, downtime for the items beinginventoried, and becomes stale shortly after the inventory is performed.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

To easily identify the discussion of any particular element or act, themost significant digit or digits in a reference number refer to thefigure number in which that element is first introduced.

FIG. 1 is a block diagram illustrating an example of a networkenvironment suitable for a system for tool mapping and inventory,according to some example embodiments.

FIG. 2 is a block diagram illustrating an example embodiment of modules(e.g., components) of a mobile device.

FIG. 3 a block diagram illustrating an example embodiment of a server.

FIG. 4 is a block diagram illustrating an example of a networkenvironment suitable for a system for tool mapping, according to someexample embodiments.

FIG. 5 is a block diagram illustrating an example of a dedicated toolboard.

FIG. 6 is a block diagram illustrating an example embodiment of modules(e.g., components) of a head mounted device.

FIG. 7 is a block diagram illustrating an example embodiment of a toolmapping module.

FIG. 8 a block diagram illustrating an example embodiment of a server.

FIG. 9 is a block diagram illustrating an example embodiment of adatabase.

FIG. 10 is a table illustrating an example of a dataset.

FIG. 11 is an interaction diagram illustrating an example ofinteractions between head mounted devices and a server.

FIG. 12 is a diagram illustrating an example of virtual contentdisplayed in a transparent display of a head mounted device.

FIG. 13 is a diagram illustrating another example of virtual contentdisplayed in a transparent display of a head mounted device.

FIG. 14 is a diagram illustrating another example of virtual contentdisplayed in a transparent display of a head mounted device.

FIG. 15 is a diagram illustrating another example of virtual contentdisplayed in a transparent display of a head mounted device.

FIG. 16 is a diagram illustrating an example of virtual contentdisplayed in a transparent display of a head mounted device pointed at adedicated tool board.

FIG. 17 is a flowchart illustrating an example operation of generatingan augmented reality content dataset at a mobile device.

FIG. 18 is a flowchart illustrating an example operation of generatingan augmented reality content dataset at a head mounted device.

FIG. 19 is a flowchart illustrating an example operation of generatingan augmented reality content dataset for a head mounted device at aserver.

FIG. 20 is a flowchart illustrating an example operation of generatingan augmented reality content dataset related to non-compliance for ahead mounted device at a server.

FIG. 21 is a flowchart illustrating an example operation of generatingan augmented reality content dataset related to an incorrect tool for ahead mounted device at a server.

FIG. 22 is a flowchart illustrating an example operation of displaying avisual indicator for alignment in a transparent display of a headmounted device at a server.

FIG. 23 is a flowchart illustrating an example operation of generatingan augmented reality content dataset related to a dedicated tool boardat a server.

FIG. 24 a block diagram illustrating components of a machine, accordingto some example embodiments, able to read instructions from amachine-readable medium and perform any one or more of the methodologiesdiscussed herein.

DETAILED DESCRIPTION

Description

Example methods and systems are directed to a live inventory systembased on sensors in multiple mobile devices at a physical location.Examples merely typify possible variations. Unless explicitly statedotherwise, structures (e.g., structural components, such as modules) areoptional and may be combined or subdivided, and operations (e.g., in aprocedure, algorithm, or other function) may vary in sequence or becombined or subdivided. In the following description, for purposes ofexplanation, numerous specific details are set forth to provide athorough understanding of example embodiments. It will be evident to oneskilled in the art, however, that the present subject matter may bepracticed without these specific details.

Maintaining an inventory of mobile tools (e.g., hammers, movableproduction or service equipment) in a large factory can be increasinglydifficult as the number of workers and tools increases. Sensors on headmounted devices may be used to identify and track the tools andequipment using computer vision, depth sensing, or other techniques. Adedicated tool board (e.g., a wall configured to store all the tools inthe factory) may be used to further keep track of the inventory of thetools. For example, the dedicated tool board may include a board on awall that is configured to store every tool to be used in a factory. Forexample, every tool has a predefined or preset storage location on theboard. The board may further include an outline of a shape of a tool soas to further guide the placement of a tool in a correct location.Therefore, a user may be able to identify the status of tools by lookingat the dedicated tool board and identifying which tools are present andwhich are not.

In one example embodiment, wearable mobile computing devices (e.g., headmounted devices) located at different locations throughout a factorysend sensor data to a server that uses the sensor data to determine andidentify which tools are present at the corresponding locations or thehead mounted devices. The server further receives an identification ofthe task selected in an augmented reality application in thecorresponding head mounted device. The task may be, for example,replacing a filter of a machine. The augmented reality applicationincludes a specific dataset that includes virtual objects to guide andinstruct the user of the head mounted device on how to perform the task(e.g., replace the filter).

The server determines which tools are required for the task beingcompleted by the user of the head mounted device. The server uses thesensor data from the head mounted devices and the identification oftools present or absent at the location of each head mounted device to:(1) identify misplaced tools; (2) identify when tools are missing fromthe dedicated tool board; (3) cross-reference tools with task managementsystems of the augmented reality applications to notify head mounteddevice users if they do not have the appropriate tools, or where to findthe appropriate tools (e.g., use wrench type H located on the dedicatedtool board, go talk to user x of head mounted device y who has thecorrect wrench type H, wrench type H is located in corner X of thewarehouse).

Furthermore, the server generates augmented reality datasets for thecorresponding head mounted devices to display the correct tool (e.g.,virtual object showing the correct tool). The server also generates adedicated tool board augmented reality dataset that displays an image ofthe user on the corresponding tool storage location on the dedicatedtool board to show which user is using the tool. Other virtual objectsmay be displayed to show which tool goes with which user. Additionalaugmented reality datasets may be used or generated to direct the userto the location of the tool.

In another example embodiment, this method can not only identify toolson a tool board, it can also identify tools in open space. This allowsfor:

-   -   Cross referencing tools in open space with tools missing on the        inventory board to notify a user when they are in close        proximity to a tool missing from the board so they can retrieve        it and place it back on the dedicated tool board.    -   Generation of 3D geometry of objects/tools based on prior 2D,        video, and/or depth captures or synthesis of any combination        thereof.    -   Being able to localize a tool in open space and direct a user to        the location of that tool based on the location of the user when        the user needs to retrieve it and take it to the correct        location, to the user assigned to a particular task that needs        that tool, or even to leave the tool in a specific spot        optimized for the next user to pick it up for a task requiring        that tool.    -   Passively identifying tools and their location without requiring        the user to actively tell the system to identify a tool or        perform inventory.

Certain example embodiments are described herein as including modules.Modules may constitute software modules (e.g., code stored or otherwiseembodied in a machine-readable medium or in a transmission medium),hardware modules, or any suitable combination thereof. A “hardwaremodule” is a tangible (e.g., non-transitory) physical component (e.g., aset of one or more processors) capable of performing certain operationsand may be configured or arranged in a certain physical manner. Invarious example embodiments, one or more computer systems or one or morehardware modules thereof may be configured by software (e.g., anapplication or portion thereof) as a hardware module that operates toperform operations described herein for that module.

In some example embodiments, a hardware module may be implementedmechanically, electronically, hydraulically, or any suitable combinationthereof. For example, a hardware module may include dedicated circuitryor logic that is permanently configured to perform certain operations. Ahardware module may be or include a special-purpose processor, such as afield programmable gate array (FPGA) or an ASIC. A hardware module mayalso include programmable logic or circuitry that is temporarilyconfigured by software to perform certain operations. As an example, ahardware module may include software encompassed within a CPU or otherprogrammable processor. It will be appreciated that the decision toimplement a hardware module mechanically, hydraulically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity that may be physically constructed,permanently configured (e.g., hardwired), or temporarily configured(e.g., programmed) to operate in a certain manner or to perform certainoperations described herein. Furthermore, as used herein, the phrase“hardware-implemented module” refers to a hardware module. Consideringexample embodiments in which hardware modules are temporarily configured(e.g., programmed), each of the hardware modules need not be configuredor instantiated at any one instance in time. For example, where ahardware module includes a CPU configured by software to become aspecial-purpose processor, the CPU may be configured as respectivelydifferent special-purpose processors (e.g., each included in a differenthardware module) at different times. Software (e.g., a software module)may accordingly configure one or more processors, for example, to becomeor otherwise constitute a particular hardware module at one instance oftime and to become or otherwise constitute a different hardware moduleat a different instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over suitable circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory (e.g., a memory device) to which itis communicatively coupled. A further hardware module may then, at alater time, access the memory to retrieve and process the stored output.Hardware modules may also initiate communications with input or outputdevices, and can operate on a resource (e.g., a collection ofinformation from a computing resource).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module in which the hardware includes one or more processors.Accordingly, the operations described herein may be at least partiallyprocessor-implemented, hardware-implemented, or both, since a processoris an example of hardware, and at least some operations within any oneor more of the methods discussed herein may be performed by one or moreprocessor-implemented modules, hardware-implemented modules, or anysuitable combination thereof.

Moreover, such one or more processors may perform operations in a “cloudcomputing” environment or as a service (e.g., within a “software as aservice” (SaaS) implementation). For example, at least some operationswithin any one or more of the methods discussed herein may be performedby a group of computers (e.g., as examples of machines that includeprocessors), with these operations being accessible via a network (e.g.,the Internet) and via one or more appropriate interfaces (e.g., anapplication program interface (API)). The performance of certainoperations may be distributed among the one or more processors, whetherresiding only within a single machine or deployed across a number ofmachines. In some example embodiments, the one or more processors orhardware modules (e.g., processor-implemented modules) may be located ina single geographic location (e.g., within a home environment, an officeenvironment, or a server farm). In other example embodiments, the one ormore processors or hardware modules may be distributed across a numberof geographic locations.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures and theirfunctionality presented as separate components and functions in exampleconfigurations may be implemented as a combined structure or componentwith combined functions. Similarly, structures and functionalitypresented as a single component may be implemented as separatecomponents and functions. These and other variations, modifications,additions, and improvements fall within the scope of the subject matterherein.

Some portions of the subject matter discussed herein may be presented interms of algorithms or symbolic representations of operations on datastored as bits or binary digital signals within a memory (e.g., acomputer memory or other machine memory). Such algorithms or symbolicrepresentations are examples of techniques used by those of ordinaryskill in the data processing arts to convey the substance of their workto others skilled in the art. As used herein, an “algorithm” is aself-consistent sequence of operations or similar processing leading toa desired result. In this context, algorithms and operations involvephysical manipulation of physical quantities. Typically, but notnecessarily, such quantities may take the form of electrical, magnetic,or optical signals capable of being stored, accessed, transferred,combined, compared, or otherwise manipulated by a machine. It isconvenient at times, principally for reasons of common usage, to referto such signals using words such as “data,” “content,” “bits,” “values,”“elements,” “symbols,” “characters,” “terms,” “numbers,” “numerals,” orthe like. These words, however, are merely convenient labels and are tobe associated with appropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “accessing,” “processing,” “detecting,” “computing,”“calculating,” “determining,” “generating,” “presenting,” “displaying,”or the like refer to actions or processes performable by a machine(e.g., a computer) that manipulates or transforms data represented asphysical (e.g., electronic, magnetic, or optical) quantities within oneor more memories (e.g., volatile memory, non-volatile memory, or anysuitable combination thereof), registers, or other machine componentsthat receive, store, transmit, or display information. Furthermore,unless specifically stated otherwise, the terms “a” or “an” are hereinused, as is common in patent documents, to include one or more than oneinstance. Finally, as used herein, the conjunction “or” refers to anon-exclusive “or,” unless specifically stated otherwise.

The following embodiments describe various example embodiments ofmethods, machine-readable media, and systems (e.g., machines, devices,or other apparatus) discussed herein.

In some embodiment, a method includes receiving, at a server, sensordata from a plurality of mobile devices having optical sensors andlocation sensors, the sensor data identifying a mobile physical objectand a location of the mobile physical object within a predefinedgeographic region based on the optical and location sensors; identifyingmobile physical objects that are present and absent within thepredefined location based on a predefined catalog of the mobile physicalobjects at the predefined geographic region; and generating, at theserver, a real-time inventory of the mobile physical objects at thepredefined location based on the mobile physical objects that arepresent and absent within the predefined location, the real-timeinventory including the identification of the mobile physical objectsand the location of the mobile physical objects within the predefinedlocation.

In some embodiments, a method may include receiving datasets from agroup of head mounted devices, each dataset identifying a task selectedin an augmented reality application of a corresponding head mounteddevice, an identification of a tool detected at the corresponding headmounted device, identifying tools present and absent at a dedicated toolboard, the dedicated tool board configured to store tools for the tasksof the augmented reality application; comparing an identification of thetools present and absent at the dedicated tool board with the toolsdetected at the head mounted devices and the tasks identified at thehead mounted devices to generate a tool inventory and a tool compliance,the tool inventory identifying tools absent from the dedicated toolboard and detected at the corresponding head mounted devices, the toolcompliance identifying whether the tool detected at the correspondinghead mounted device is valid for the task selected in the augmentedreality application of the corresponding head mounted device; generatingan augmented reality content dataset for each head mounted device, eachaugmented reality content dataset; and/or generating a dedicated toolboard augmented reality content dataset for the dedicated tool boardbased on the tool inventory.

In some embodiments, the generating an augmented reality content datasetfor each head mounted device, each augmented reality content dataset mayinclude a virtual object identifying at least one of a missing tool, anincorrect tool, and a valid tool based on the tool compliance.

In some embodiments, the dedicated tool board augmented reality contentdataset may include a group of virtual objects identifying users of thehead mounted devices with corresponding tools on the dedicated toolboard.

In some embodiments, the dataset may further include an identificationof a user for each head mounted device, an identification of a physicalobject within a field of view of each head mounted device.

In some embodiments, the task identifying a physical operation toperform on the physical object.

In some embodiments, the augmented reality application configured togenerate virtual objects displayed in a transparent display of thecorresponding head mounted device.

In some embodiments, the virtual objects may include a visualillustration of how to perform the task and how operate the tool relatedto the task on the physical object.

In some embodiments, the dedicated tool board may include a group ofoutlines displayed on the dedicated tool board, each outlinecorresponding to a tool on the dedicated tool board.

In some embodiments, such a method may further include using a depthsensor of a head mounted device with the dedicated tool board within afield of view of the head mounted device, to determine the tools presentat the dedicated tool board and/or identifying tools absent from thededicated tool board based depth sensor data, a shape of the outline,and a location of the outline relative to the dedicated tool board.

In some embodiments, the virtual object may include a three-dimensionalmodel of a tool related to the task selected at the corresponding headmounted device.

In some embodiments, the virtual object may include a visual indicatorto validate a tool detected at the corresponding head mounted device forthe task selected at the corresponding head mounted device.

In some embodiments, the virtual object may include a visual indicatorto identify the tool detected at the corresponding head mounted deviceas an incorrect tool for the task selected at the corresponding headmounted device.

In some embodiments, the dedicated tool board augmented reality contentdataset may include, at least one of: a first virtual object identifyinga user of a head mounted device corresponding to a tool absent from thededicated tool board, a second virtual object, and/or a third virtualobject identifying a user of a head mounted device corresponding to atool present on the dedicated too board.

In some embodiments, the second virtual object may include a visualindicator linking a tool present on the dedicated tool board to thefirst virtual object.

In some embodiments, such a method may further include communicating theaugmented reality content dataset to the corresponding head mounteddevice and/or causing a display of the augmented reality content datasetin a transparent display of the corresponding head mounted device.

In some embodiments, such a method may further include communicating thededicated tool board augmented reality content dataset to a head mounteddevice with the dedicated tool board within a field of view of the headmounted device and/or causing a display of the dedicated tool boardaugmented reality content dataset in a transparent display of the headmounted device with the dedicated tool board within the field of view ofthe head mounted device.

DRAWINGS

FIG. 1 is a block diagram illustrating an example of a networkenvironment suitable for a system for tool mapping and inventory,according to some example embodiments.

A network environment 100 includes mobiles 104, 110, and 116, externalsensors 126, and a server 124, communicatively coupled to each other viaa network 120. The mobile devices 104, 110, 116 and the server 124 mayeach be implemented in a computer system, in whole or in part, asdescribed below with respect to FIG. 24.

The server 124 may be part of a network-based system. For example, thenetwork-based system may be or include a cloud-based server system thatprovides additional information, such as physical tool inventory,virtual objects (e.g., 3D model), tool identification, tool locationwithin a predefined geographic location 102, to the mobile devices 104,110, 116. The server 124 receives sensor data from the mobile device104, 110, 116 to identify tools present within the location 102. Theserver 124 generates an inventory of tools present and absent at thelocation 102 based on the sensor data from the mobile devices 104, 110,116 and a predefined list of tools associated with the location 102and/or the users 108, 114, 122. The inventory may include anidentification of tools present and absent from the location 102, alocation of the tools present at the location 102, users presently usingthe tools or associated with the tools at the location 102.

The mobile devices 104, 112, 116 each include a computing device and adisplay (e.g., a transparent display) that displays syntheticinformation in a layer added onto a field of view of the users 114, 122,124. For example, the user 108 may aim the mobile device 104 and look ata physical machine 106 (e.g., a drill) in a real world physicalenvironment (e.g., factory at location 102). The user 108 uses themobile device 104 to view the machine 112 and a physical tool 128 (e.g.,screwdriver). The physical tool may include a physical object notcapable of communicating electronically with other computing devicessuch as the server 124 or the mobile device 104. The user 108, 114, 122may be a human user (e.g., a human being), a machine user (e.g., acomputer configured by a software program to interact with the mobiledevice 104, 110, 116), or any suitable combination thereof (e.g., ahuman assisted by a machine or a machine supervised by a human). Theuser 108, 114, 122 is not part of the network environment 100, but isassociated with the mobile device 104, 110, 116. For example, the mobiledevice 104, 110, 116 may be a computing device with a camera and adisplay such as a tablet, smartphone, or a wearable computing device(e.g., head mounted device such as a helmet or glasses). In anotherexample embodiment, the computing device may be hand held or may beremovably mounted to the head of the user 108, 114, 122. In one example,the display may be a screen that displays what is captured with a cameraof the mobile device 104, 112, 116. In another example, the display ofthe mobile device 104, 110, 116 may be transparent such as in lenses ofwearable computing glasses or the visor or a face shield of a helmet.

In one example embodiment, the objects in the image generated by themobile devices are tracked and recognized locally at the mobile devicesusing a local context recognition dataset or any other previously storeddataset of an augmented reality application of the mobile devices. Forexample, the local context recognition dataset module may include alibrary of virtual objects associated with real-world physical objectsor references. In one example, the mobile device 104 identifies featurepoints in an image of the machine 106 and the tool 118. The mobiledevice 104 may also identify tracking data related to the machine 106(e.g., location 102, GPS location of the mobile device 104, orientation,distance to the machine 106). If the captured image is not recognizedlocally at the mobile device 104, the mobile device 104 can downloadadditional information (e.g., 3D model or other augmented data)corresponding to the captured image, from a database of the server 124over the network 120.

In another example embodiment, the machine 106 in the image generated bythe mobile device 104 is tracked and recognized remotely at the server124 using a remote context recognition dataset or any other previouslystored dataset of an AR application in the server 124. The remotecontext recognition dataset module may include a library of virtualobjects or augmented information associated with real-world physicalobjects or references.

External sensors 126 may be associated with, or coupled to, related tothe machines 106, 112 to measure a location, status, and characteristicsof the machines 106, 112. Examples of measured readings may include andbut are not limited to tool presence, weight, pressure, temperature,velocity, direction, position, intrinsic and extrinsic properties,acceleration, and dimensions. For example, external sensors 126 may bedisposed throughout a factory floor (e.g., location 102) to measuremovement, pressure, orientation, and temperature. The external sensors126 can also be used to measure a location, status, and characteristicsof the mobile device 104, 110, 116. The server 124 can compute readingsfrom data generated by the external sensors 408 and generate virtualindicators such as vectors or colors based on data from external sensors408. Virtual indicators are then overlaid on top of a live image or aview of the machine 106, 112 in a line of sight of the correspondinguser to show data related to the machine 106, machine 112, tool 118,tool 128. For example, the virtual indicators may include arrows withshapes and colors that change based on real-time data. The mobile device104, 110, 116 can render the virtual indicators in the display of themobile devices. In another example embodiment, the virtual indicatorsare rendered at the server 124 and streamed to the mobile devices 104,110, 116.

The external sensors 126 may include other sensors used to track thelocation, movement, and orientation of the mobile devices 104, 110, 116externally without having to rely on sensors internal to the mobiledevices. The external sensors 126 may include optical sensors (e.g.,depth sensors such as structure light, time of flight), wireless sensors(Bluetooth, Wi-Fi), GPS sensors, and audio sensors to determine thelocation of the users 108, 114, 122, distance of the users 108, 114, 122to the external sensors 126 (e.g., sensors placed in corners of a venueor a room), the orientation of the mobile devices 104, 110, 116 to trackwhat the user 108 is looking at (e.g., direction at which the mobiledevice 104 is pointed).

In another example embodiment, data from the external sensors 408 andinternal sensors in the mobile device 104 may be used for analytics dataprocessing at the server 124 (or another server) for analysis on usageand how the user 108 is interacting with the machine 106 in the physicalenvironment. Live data from other servers may also be used in theanalytics data processing. For example, the analytics data may track atwhat locations (e.g., points or features) on the physical or virtualobject the user 108 has looked, how long the user 108 has looked at eachlocation on the physical or virtual object, how the user 108 positionsthe mobile device 104 when looking at the physical or virtual object,which features of the virtual object the user 108 interacted with (e.g.,such as whether the user 108 engaged with the virtual object), and anysuitable combination thereof. The mobile device 104 receives avisualization content dataset related to the analytics data. The mobiledevice 104 then generates a virtual object with additional orvisualization features, or a new experience, based on the visualizationcontent dataset.

Any of the machines, databases, or devices shown in FIG. 1 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIG. 24. As used herein, a “database” is a data storage resource and maystore data structured as a text file, a table, a spreadsheet, arelational database (e.g., an object-relational database), a triplestore, a hierarchical data store, or any suitable combination thereof.Moreover, any two or more of the machines, databases, or devicesillustrated in FIG. 1 may be combined into a single machine, and thefunctions described herein for any single machine, database, or devicemay be subdivided among multiple machines, databases, or devices.

The network 120 may be any network that enables communication between oramong machines (e.g., server 124), databases, and devices (e.g., mobiledevices 104, 110, 116). Accordingly, the network 120 may be a wirednetwork, a wireless network (e.g., a mobile or cellular network), or anysuitable combination thereof. The network 120 may include one or moreportions that constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof.

The mobile device 104 includes sensors 204, a display 206, a processor210, and a storage device 208. For example, the mobile device 104 mayinclude a computing device such as a smart phone or a tablet.

The sensors 204 include, for example, a thermometer, an infrared camera,a barometer, a humidity sensor, an EEG sensor, a proximity or locationsensor (e.g, near field communication, GPS, Bluetooth, Wifi), an opticalsensor (e.g., camera), an orientation sensor (e.g., gyroscope), an audiosensor (e.g., a microphone), or any suitable combination thereof. Forexample, the sensors 204 may include a rear facing camera and a frontfacing camera in the mobile device 104. It is noted that the sensorsdescribed herein are for illustration purposes and the sensors 204 arethus not limited to the ones described.

The display 206 includes, for example, a display configured to displayimages generated by the processor 210. In another example, the display206 includes a touch sensitive surface to receive a user input via acontact on the touch sensitive surface.

The processor 210 includes a tool recognition module 202, and a toolinventory module 212. The tool recognition module 202 receives data fromsensors 204 (e.g., receive an image of the machine 106 or the tool 118)and identifies and recognizes the machine 106/tool 118 usingmachine-vision recognition techniques. The tool recognition module 202then retrieves from the storage device 208 content associated with themachine 106 and/or tool 118. In one example embodiment, the toolrecognition module 202 identifies a visual reference (e.g., a logo or QRcode) on the physical object (e.g., a chair) and tracks the location ofthe visual reference. The visual reference may also be referred to as amarker and may consist of an identifiable image, symbol, letter, number,machine-readable code disposed on the tool 118. For example, the visualreference may include a bar code, a quick response (QR) code, or animage that has been previously associated with the virtual object.

The tool inventory module 212 generates a tool inventory based on datafrom sensor 204 and an identification of the tools from the toolrecognition module 202. For example, the tool inventory module 212collects information related to the type of tools used at the location102, the users associated with the tools, and the location of the toolsrecognized by the mobile device 104 within the location 102.Furthermore, the tool inventory module 212 compares the tools identifiedat the location 102 with a list of predefined tools for the location102. For example, the list of predefined tools identifies physical toolsthat should be present (or absent) at the location 102, specificlocations or machines associated with the physical tools, specific usersassociated with the tools or authorized to use certain tools, a time atwhich a physical tool should be present in a particular location (e.g.,next to machine 106 or with a pre-specified user 108) within thelocation 102. The tool inventory module 212 may identify tools that areabsent based on the list of predefined tools for the location 102.

The tool inventory module 212 dynamically updates an inventory of thetools present at the location 102 based on sensor data from the mobiledevice 104 and other mobile devices at the same location 102. Forexample, sensor data from the mobile devices may be collected at theserver 124 to generate a real-time inventory of the tools present at thelocation 102.

The storage device 208 stores an identification of the sensors and theirrespective functions. The storage device 208 further includes a databaseof visual references (e.g., images, visual identifiers, features ofimages) and corresponding metadata (e.g., which user are allowed to usethe tool, where the tool should be located within the location 102 atcertain time periods, which machine within the location 102 isassociated with the tool, etc). For example, the visual reference mayinclude a machine-readable code or a previously identified image (e.g.,a picture of a screwdriver).

In one embodiment, the mobile device 104 communicates over the network120 with the server 124 to retrieve a portion of a database of visualreferences and corresponding metadata. The network 120 may be anynetwork that enables communication between or among machines, databases,and devices (e.g., the mobile device 104). Accordingly, the network 120may be a wired network, a wireless network (e.g., a mobile or cellularnetwork), or any suitable combination thereof. The network 120 mayinclude one or more portions that constitute a private network, a publicnetwork (e.g., the Internet), or any suitable combination thereof.

Any one or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aprocessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

FIG. 3 is a block diagram illustrating an example embodiment of aserver. The server 124 includes an external sensor interface module 304,a mobile device interface module 306, a processor 302, and a database312.

The external sensor interface module 304 is configured to communicatewith the external sensors 108 to receive sensor data related to themobile devices and the location 102. For example, the external sensorinterface module 304 accesses data related to the presence and absenceof the tools at the location 102.

The mobile device interface module 306 is configured to communicate withthe mobile device 104, 110, 116 located within the machine 112 toreceive data identifying a machine and a tool detected at the mobiledevice 104, a user identification of the mobile device 104, and ageographic location of the mobile device 104.

The processor 302 includes a server inventory application 308 and aserver compliance application 310. The server inventory application 308performs a real time inventory of the tools based on the data receivedfrom the external sensor interface module 304 and the mobile deviceinterface module 306. For example, the server inventory application 308tracks the location of tools detected within the location 102. Theserver inventory application 308 associates the identification of eachtool with their corresponding location (e.g., screwdriver type B is withmobile device 110, wrench type C is with mobile device 116).

The server compliance application 310 determines a compliance of themobile device 104, 110, 116 based on their respective tasks, location,user identification, tool(s) detected at the corresponding mobiledevice. The server compliance application 310 determines whether thetool detected at each mobile device 104 matches the tool specified orassociated with a task at the mobile device. For example, if a task forthe mobile device 104 includes changing a filter of a machine x, thetool associated with that task may be a type A wrench. The servercompliance application 310 detects that the user of the mobile device104 has in his possession a type B wrench instead of the type A wrenchand generates AR content within a field of view of the user 108 to warnthe user 108 that he/she has the incorrect tool. The AR content mayfurther identify where to find the correct tool (e.g., another user hasit, or the tool can be found at a specific location identified by one ofthe mobile devices within location 102).

The database 312 stores data received from the external sensor interfacemodule 304 and the mobile device interface module 306, and predefinedtools associated with predefined tasks. The database 312 may keep a liveor real-time inventory of the location of the tools, which tool isassociated with which mobile device, and which tool is associated withwhich task.

FIG. 4 is a block diagram illustrating an example of a networkenvironment suitable for a system for tool mapping, according to someexample embodiments.

A network environment 400 includes head mounted devices 406, 412, and416, external sensors 408, and a server 402, communicatively coupled toeach other via a network 428. The head mounted devices 406, 412, 416 andthe server 402 may each be implemented in a computer system, in whole orin part, as described below with respect to FIG. 24.

The server 402 may be part of a network-based system. For example, thenetwork-based system may be or include a cloud-based server system thatprovides additional information, such as 3D models or other virtualobjects, to the head mounted devices 406, 412, 416.

The head mounted devices 406, 412, 416 each include a computing deviceand a transparent display that displays synthetic information in a layeradded onto a field of view of the users 414, 422, 424. For example, auser 414 wear the head mounted device 406 and look at a machine 418(e.g., a drill) in a real world physical environment (e.g., factory atlocation 410). A user 422 uses the head mounted device 412 to view themachine 420 and a tool 426 e.g., screwdriver). A user 424 uses the headmounted device 416 to view a dedicated tool board 404. The dedicatedtool board 404 may be a central storage facility for all tools withinthe location 410. In one example embodiment, users 414, 422, 424 abideby a system or standard that all tools used within the location 410 areto be stored backed at the dedicated tool board 404 after their use.

The users 414, 422, 424 may be a human user (e.g., a human being), amachine user (e.g., a computer configured by a software program tointeract with the head mounted device 406, 412, 416), or any suitablecombination thereof (e.g., a human assisted by a machine or a machinesupervised by a human). The users 414, 422, 424 are not part of thenetwork environment 400, but are associated with the head mounted device406, 412, 416. For example, the head mounted device 406 may be acomputing device with a camera and a transparent display such as awearable computing device (e.g., helmet or glasses). In another exampleembodiment, the computing device may be hand held or may be removablymounted to the head of the users 414, 422, and 424. In one example, thedisplay in the head mounted device 406 may be a screen that displayswhat is captured with a camera of the head mounted device 406, 412, 416.In another example, the display of the head mounted device 406, 412, 416may be transparent such as in lenses of wearable computing glasses orthe visor or a face shield of a helmet.

The user 414 may be a user of an AR application in the head mounteddevice 406 and at the server 402. The AR application may provide theuser 414 with an AR experience triggered by identified objects (e.g.,machine 418) in the physical environment. For example, the machine 418includes identifiable objects such as a 2D physical object (e.g., apicture), a 3D physical object (e.g., a factory machine), a location(e.g., at the bottom floor of a factory), or any references (e.g.,perceived corners of walls or furniture) in the real world physicalenvironment. The AR application may include computer vision recognitionto determine corners, objects, lines, letters, etc.

The AR applications allow a user to experience information, such as inthe form of a virtual object (e.g., a three-dimensional model of avirtual dinosaur) overlaid on an image of a real world physical object(e.g., a billboard) captured by a camera of a viewing device. Theviewing device may include a handheld device such as a tablet orsmartphone, or a wearable device such as a head mounted device (HMD)(e.g., helmet, glasses). The virtual object may be displayed in atransparent or clear display (e.g., see-through display) of the viewingdevice. The physical object may include a visual reference (e.g.,uniquely identifiable pattern on a physical object) that the ARapplication can recognize. A visualization of the additionalinformation, such as the virtual object overlaid or engaged with animage of the physical object is generated in the display of the viewingdevice. The viewing device generates the virtual object based on therecognized visual reference (e.g., QR code) or captured image of thephysical object (e.g, image of a logo). The viewing device displays thevirtual object based on a relative position between the viewing deviceand the visual reference. For example, a virtual dinosaur appears closerand bigger when the viewing device is held closer to the visualreference associated with the virtual dinosaur. Similarly, the virtualdinosaur appears smaller and farther when the viewing device is movedfurther away from the virtual reference associated with the virtualdinosaur. The virtual object may include a three-dimensional model of avirtual object or a two-dimensional model of a virtual object. Forexample, the three-dimensional model includes a three-dimensional viewof a chair. The two-dimensional model includes a two-dimensional view ofa dialog box, menu, or written information such as statisticsinformation for a baseball player. The viewing device renders an imageof the three-dimensional or two-dimensional model of the virtual objectin the display of the viewing device.

In one example embodiment, the AR application includes several tasks oroperations for the user to perform with respect to the machine 418. Forexample, one task may be cleaning the machine 418. Another task may bechanging a component (e.g., filter) of the machine 418. The user 414 mayselect the task from a menu of task from the AR application. In anotherexample, the task may be already pre-assigned to the user 414. Forexample, the server 402 assigns a task of cleaning the machine 418 tothe user 414 every Monday morning or upon other configurable triggers.

In one example embodiment, the objects in the image are tracked andrecognized locally in the head mounted device 406 using a local contextrecognition dataset or any other previously stored dataset of the ARapplication of the head mounted device 406. The local contextrecognition dataset module may include a library of virtual objectsassociated with real-world physical objects or references. In oneexample, the head mounted device 412 identifies feature points in animage of the machine 420 and the tool 426. The head mounted device 412may also identify tracking data related to the machine 420 (e.g.,location 410, GPS location of the head mounted device 412, orientation,distance to the machine 420). If the captured image is not recognizedlocally at the head mounted device 412, the head mounted device 412 candownload additional information (e.g., 3D model or other augmented data)corresponding to the captured image, from a database of the server 402over the network 428.

In another example embodiment, the machine 420 in the image is trackedand recognized remotely at the server 402 server 124 using a remotecontext recognition dataset or any other previously stored dataset of anAR application in the server 124. The remote context recognition datasetmodule may include a library of virtual objects or augmented informationassociated with real-world physical objects or references.

Similarly, the head mounted device 416 identifies feature points in animage of the dedicated tool board 404 to identify which tools arepresent and which tools are missing from the dedicated tool board 404.The head mounted device 416 may use other techniques such as 3D mappingand depth sensing to map the tools physically present on the dedicatedtool board 404. Other techniques (e.g., machine-vision algorithm) can beused to identify which tools are present on the dedicated tool board404.

External sensors 408 may be associated with, coupled to, related to themachines 418, 420, and dedicated tool board 404 to measure a location,status, and characteristics of the machines 418, 420 and dedicated toolboard 404. Examples of measured readings may include and but are notlimited to tool presence, weight, pressure, temperature, velocity,direction, position, intrinsic and extrinsic properties, acceleration,and dimensions. For example, external sensors 408 may be disposedthroughout a factory floor (e.g., location 410) to measure movement,pressure, orientation, and temperature. The external sensors 408 canalso be used to measure a location, status, and characteristics of thehead mounted device 406, 412, 416. The server 402 can compute readingsfrom data generated by the external sensors 408 and generate virtualindicators such as vectors or colors based on data from external sensors408. Virtual indicators are then overlaid on top of a live image or aview of the machine 418, 420, and dedicated tool board 404 in a line ofsight of the corresponding user to show data related to the machines418, 420, tool 426, and dedicated tool board 404. For example, thevirtual indicators may include arrows with shapes and colors that changebased on real-time data. The head mounted device 406, 412, 416 canrender the virtual indicators in the transparent display of thecorresponding head mounted device. In another example embodiment, thevirtual indicators are rendered at the server 402 and streamed to thehead mounted device 406, 412, 416.

The external sensors 408 may include other sensors used to track thelocation, movement, and orientation of the head mounted device 406, 412,416 externally without having to rely on sensors internal to the headmounted device 406, 412, 416. The sensors may include optical sensors(e.g., depth-enabled 3D camera), wireless sensors (Bluetooth, Wi-Fi),GPS sensors, and audio sensors to determine the location of the users414, 422, 424 wearing the head mounted device 406, 412, 416, distance ofthe user 114 to the external sensors 408 (e.g., sensors placed incorners of a venue or a room), the orientation of the head mounteddevice 406 to track what the user 414 is looking at (e.g., direction atwhich the head mounted device 406 is pointed, e.g., head mounted device406 pointed towards a player on a tennis court, head mounted device 406pointed at a person in a room).

In another example embodiment, data from the external sensors 408 andinternal sensors in the head mounted device 406 may be used foranalytics data processing at the server 402 (or another server) foranalysis on usage and how the user 414 is interacting with the machine418 in the physical environment. Live data from other servers may alsobe used in the analytics data processing. For example, the analyticsdata may track at what locations (e.g., points or features) on thephysical or virtual object the user 414 has looked, how long the user414 has looked at each location on the physical or virtual object, howthe user 414 wore the head mounted device 406 when looking at thephysical or virtual object, which features of the virtual object theuser 414 interacted with (e.g., such as whether the user 414 engagedwith the virtual object), and any suitable combination thereof. The headmounted device 406 receives a visualization content dataset related tothe analytics data. The head mounted device 406 then generates a virtualobject with additional or visualization features, or a new experience,based on the visualization content dataset.

Any of the machines, databases, or devices shown in FIG. 4 may beimplemented in a general-purpose computer modified (e.g., configured orprogrammed) by software to be a special-purpose computer to perform oneor more of the functions described herein for that machine, database, ordevice. For example, a computer system able to implement any one or moreof the methodologies described herein is discussed below with respect toFIG. 24. As used herein, a “database” is a data storage resource and maystore data structured as a text file, a table, a spreadsheet, arelational database (e.g., an object-relational database), a triplestore, a hierarchical data store, or any suitable combination thereof.Moreover, any two or more of the machines, databases, or devicesillustrated in FIG. 1 may be combined into a single machine, and thefunctions described herein for any single machine, database, or devicemay be subdivided among multiple machines, databases, or devices.

The network 428 may be any network that enables communication between oramong machines (e.g., server 402) databases, and devices (e.g., headmounted device 406, 412, 416). Accordingly, the network 428 may be awired network, a wireless network (e.g., a mobile or cellular network),or any suitable combination thereof. The network 428 may include one ormore portions that constitute a private network, a public network (e.g.,the Internet), or any suitable combination thereof.

FIG. 5 is a block diagram illustrating an example of a dedicated toolboard 404. The dedicated tool board 404 may be, for example, a physicalwall having preset storage locations for the tools used at the location410 (e.g., within a factory). Outlines or cutout shapes of the tools mayvisually represent where each tool is to be placed on the dedicated toolboard 404.

For example, the dedicated tool board 404 includes tools (e.g., tools502, 504, 510, and 514) that are physically placed on a vertical storagewall. Therefore tools 502, 504, 510, and 514 are present on thededicated tool board 404. Tools that are absent or missing from thededicated tool board 404 are identified with an outline (e.g., outlines506, 508, and 512) or a contour shape of the respective tool on thededicated tool board 404.

In another example embodiment, sensors may be placed on the dedicatedtool board 404 to detect the presence or absence of specific tools basedon their preset locations on the dedicated tool board 404.

FIG. 6 is a block diagram illustrating an example embodiment of modules(e.g., components) of a head mounted device 406.

The head mounted device 406 includes sensors 602, a transparent display614, a processor 606, and a storage device 604. For example, the headmounted device 406 may include a helmet, a visor, or any other devicemounted to a head for the user 414.

The sensors 602 include, for example, a thermometer, an infrared camera,a barometer, a humidity sensor, an EEG sensor, a proximity or locationsensor (e.g, near field communication, GPS, Bluetooth, Wifi), an opticalsensor (e.g., camera), an orientation sensor (e.g., gyroscope), an audiosensor (e.g., a microphone), or any suitable combination thereof. Forexample, the sensors 602 may include a rear facing camera and a frontfacing camera in the head mounted device 406. It is noted that thesensors described herein are for illustration purposes and the sensors602 are thus not limited to the ones described.

The transparent display 614 includes, for example, a display configuredto display images generated by the processor 606. The user 414 can seethrough the transparent display 614. Images displayed in the transparentdisplay 614 appear to be perceived as a layer on real world physicalobjects by the user 414.

The processor 606 includes an AR application 608, a rendering module610, and a tool mapping and localization module 612. The AR application608 receives data from sensors 602 (e.g., receive an image of themachine 418 or a physical tool) and identifies and recognizes themachine 418 using machine-vision recognition techniques. The ARapplication 608 then retrieves from the storage device 604 AR contentassociated with the machine 418. In one example embodiment, the ARapplication 608 identifies a visual reference (e.g., a logo or QR code)on the physical object (e.g., a chair) and tracks the location of thevisual reference within the transparent display 614 of the head mounteddevice 406. The visual reference may also be referred to as a marker andmay consist of an identifiable image, symbol, letter, number,machine-readable code. For example, the visual reference may include abar code, a quick response (QR) code, or an image that has beenpreviously associated with the virtual object.

The rendering module 610 renders virtual objects based on data fromsensors 602. For example, the rendering module 610 renders a display ofa virtual object (e.g., a door with a color based on the temperatureinside the room as detected by sensors from HMDs inside the room) basedon a three-dimensional model of the virtual object (e.g., 3D model of adrill) associated with the machine 418 (e.g., a drill). In anotherexample, the rendering module 610 generates a display of the virtualobject overlaid on an image of the machine 418 captured by a camera ofthe head mounted device 406. The virtual object may be furthermanipulated (e.g., by the user 414) by moving the machine 418 relativeto the head mounted device 406. Similarly, the display of the virtualobject may be manipulated (e.g., by the user 414) by moving the headmounted device 406 relative to the machine 418.

In another example embodiment, the rendering module 610 includes a localrendering engine that generates a visualization of a three-dimensionalvirtual object overlaid (e.g., superimposed upon, or otherwise displayedin tandem with) on an image of the machine 418 captured by a camera ofthe head mounted device 406 or a view of the physical object in thetransparent display 614 of the head mounted device 406. A visualizationof the three-dimensional virtual object may be manipulated by adjustinga position of the machine 418 (e.g., its physical location, orientation,or both) relative to the camera of the head mounted device 406.Similarly, the visualization of the three-dimensional virtual object maybe manipulated by adjusting a position camera of the head mounted device406 relative to the machine 418.

In one example embodiment, the rendering module 610 identifies themachine 418 (e.g., a physical telephone) based on data from sensors 602and external sensors 408, accesses virtual functions (e.g., increase orlower the volume of a nearby television) associated with physicalmanipulations (e.g., lifting a physical telephone handset) of themachine 418, and generates a virtual function corresponding to aphysical manipulation of the machine 418.

In another example embodiment, the rendering module 610 determineswhether the captured image matches an image locally stored in thestorage device 604 that includes a local database of images andcorresponding additional information (e.g., three-dimensional model andinteractive features). The rendering module 610 retrieves a primarycontent dataset from the server 402, generates and updates a contextualcontent dataset based on an image captured with the head mounted device406.

The tool mapping and localization module 612 generates a virtual objector a visual indicator to be displayed in the transparent display 614.The virtual object may include, for example, a renderedthree-dimensional model of a tool (e.g., screwdriver) to be used withthe machine 418 in connection with performing a task of the ARapplication 608. For example, the tool mapping and localization module612 causes a display of a screw driver perceived as hovering above themachine 418. In another example, the tool mapping and localizationmodule 612 determines that a present tool detected at the head mounteddevice 406 is an incorrect tool. The tool mapping and localizationmodule 612 causes a display of a virtual cross perceived on top of theincorrect tool.

The storage device 604 stores an identification of the sensors and theirrespective functions. The storage device 604 further includes a databaseof visual references (e.g., images, visual identifiers, features ofimages) and corresponding experiences (e.g., three-dimensional virtualobjects, interactive features of the three-dimensional virtual objects).For example, the visual reference may include a machine-readable code ora previously identified image (e.g., a picture of a screwdriver). Thepreviously identified image of the screwdriver may correspond to athree-dimensional virtual model of the screwdriver that can be viewedfrom different angles by manipulating the position of the head mounteddevice 406 relative to the picture of the screwdriver. Features of thethree-dimensional virtual screwdriver may include selectable icons onthe three-dimensional virtual model of the screwdriver. An icon may beselected or activated using a user interface on the head mounted device406.

In another example embodiment, the storage device 604 includes a primarycontent dataset, a contextual content dataset, and a visualizationcontent dataset. The primary content dataset includes, for example, afirst set of images and corresponding experiences (e.g., interactionwith three-dimensional virtual object models). For example, an image maybe associated with one or more virtual object models. The primarycontent dataset may include a core set of images of the most popularimages determined by the server 402. The core set of images may includea limited number of images identified by the server 402. For example,the core set of images may include the images depicting covers of theten most popular machines and their corresponding experiences (e.g.,virtual objects that represent the ten most popular machines). Inanother example, the server 402 may generate the first set of imagesbased on the most popular or often scanned images received at the server402. Thus, the primary content dataset does not depend on objects orimages scanned by the rendering module 610 of the head mounted device406.

The contextual content dataset includes, for example, a second set ofimages and corresponding experiences (e.g., three-dimensional virtualobject models) retrieved from the server 402. For example, imagescaptured with the head mounted device 406 that are not recognized (e.g.,by the server 402) in the primary content dataset are submitted to theserver 402 for recognition. If the captured image is recognized by theserver 402, a corresponding experience may be downloaded at the headmounted device 406 and stored in the contextual content dataset. Thus,the contextual content dataset relies on the context in which the headmounted device 406 has been used. As such, the contextual contentdataset depends on objects or images scanned by the rendering module610.

In one embodiment, the head mounted device 406 may communicate over thenetwork 428 with the server 402 to retrieve a portion of a database ofvisual references, corresponding three-dimensional virtual objects, andcorresponding interactive features of the three-dimensional virtualobjects. The network 428 may be any network that enables communicationbetween or among machines, databases, and devices (e.g., the headmounted device 406). Accordingly, the network 428 may be a wirednetwork, a wireless network (e.g., a mobile or cellular network), or anysuitable combination thereof. The network 428 may include one or moreportions that constitute a private network, a public network (e.g., theInternet), or any suitable combination thereof.

Any one or more of the modules described herein may be implemented usinghardware (e.g., a processor of a machine) or a combination of hardwareand software. For example, any module described herein may configure aprocessor to perform the operations described herein for that module.Moreover, any two or more of these modules may be combined into a singlemodule, and the functions described herein for a single module may besubdivided among multiple modules. Furthermore, according to variousexample embodiments, modules described herein as being implementedwithin a single machine, database, or device may be distributed acrossmultiple machines, databases, or devices.

FIG. 7 is a block diagram illustrating an example embodiment of a toolmapping module.

The tool mapping and localization module 612 includes a machinedetection module 702, a tool detection module 704, an ar task detectionmodule 706, a device inventory module 710, and a tool localizationmodule 712.

The machine detection module 702 is configured to detect and identifythe machine 418 using a combination of techniques such as location ofthe machine 418, machine-vision to identify the machine 418, othervisual and non-visual indicator to uniquely identify the machine 418.

The tool detection module 704 is configured to detect and identify atool present at the machine 418 and at the head mounted device 406. Forexample, the tool detection module 704 determines the presence of thetool by using machine-vision technique or using depth sensors to map thephysical characteristics of the tool and identify the tool based on thedepth sensor data. Other techniques may include determining the shape ofthe tool to identify the type of tool. The tool detection module 704 mayalso use visual indicators (e.g., QR code, serial numbers) on the toolto identify the tool.

The ar task detection module 706 determines the task associated with theAR application 608. For example, the task may include cleaning a filterof the machine 418. The task may be selected by the user 414 of the headmounted device 406 or may be assigned to the user 414 by the server 402.

The device inventory module 710 is configured to communicate with theserver 402 to access a real time inventory of the tools at the location410. For example, the device inventory module 710 accesses informationgenerated by the tool detection module 704 and communicate whether atool is detected and present at the head mounted device 406 to theserver 402. In another example, the device inventory module 710 receivesinventory data from the server 402 indicating and identifying whichtools are present and absent at the dedicated tool board 404, and whichhead mounted device or user has the missing tool from the dedicated toolboard 404. The tool localization module 712 receives inventory data fromthe server 402 indicating and identifying which tools are present andabsent at the dedicated tool board 404. The tool localization module 712further identifies the location of missing tools from the dedicated toolboard 404 based on the inventory data. For example, the toollocalization module 712 determines that a hammer is located with theuser of the head mounted device 412.

The device compliance module 708 determines whether the tool detected atthe head mounted device 412 matches the tool specified or associatedwith the task at the head mounted device 412. For example, if a taskincludes changing a filter of a machine x, the tool associated with thattask may be a type A wrench. The device compliance module 708 detectsthat the user of the head mounted device 412 has in his possession atype B wrench instead of the type A wrench and generates AR contentwithin a field of view of the user 422 to warn the user 422 that he/shehas the incorrect tool. The AR content may further identify where tofind the correct tool within the location 410.

FIG. 8 is a block diagram illustrating an example embodiment of aserver. The server 402 includes an external sensor interface module 802,a head mounted display interface module 804, a processor 814, and adatabase 810.

The external sensor interface module 802 is configured to communicatewith the external sensors 408 to receive sensor data related to the headmounted devices, the location 410, and the dedicated tool board 404. Forexample, the external sensor interface module 802 accesses presence datarelated to tools on the dedicated tool board 404.

The head mounted display interface module 804 is configured tocommunicate with the head mounted devices 404, 412, 416 located withinthe location 410 to receive data identifying a machine and a tooldetected at the head mounted device, a task of the AR application in thecorresponding head mounted device, a user identification of the headmounted device, and a location of the head mounted device.

The processor 814 includes a server inventory application 806 and aserver compliance application 808. The server inventory application 806performs a real time inventory of the tools based on the data receivedfrom the external sensor interface module 802 and the head mounteddisplay interface module 804. For example, the server inventoryapplication 806 tracks the location of each tool from the dedicated toolboard 404. The server inventory application 806 associates theidentification of each tool with their corresponding location (e.g.,screwdriver type B is with head mounted device 412, wrench type C ispresent on the dedicated tool board 404).

The server compliance application 808 determines a compliance of thehead mounted devices based on their respective tasks, location, useridentification, tool(s) detected at the corresponding head mounteddevice. Similarly to device compliance module 708, the server complianceapplication 808 determines whether the tool detected at each headmounted device matches the tool specified or associated with the task atthe corresponding head mounted device. For example, if a task for headmounted device 412 includes changing a filter of a machine x, the toolassociated with that task may be a type A wrench. The server complianceapplication 808 detects that the user of the head mounted device 412 hasin his possession a type B wrench instead of the type A wrench andgenerates AR content within a field of view of the user 422 to warn theuser 422 that he/she has the incorrect tool. The AR content may furtheridentify where to find the correct tool (e.g., another user has it, orthe tool can be found on the dedicated tool board 404).

The database 810 stores data received from the external sensor interfacemodule 802 and the head mounted display interface module 804, andpredefined tools associated with predefined tasks. The database 810 maykeep a live inventory of the location of the tools, which tool isassociated with which head mounted device, which tool is associated withwhich task.

FIG. 9 is a block diagram illustrating an example embodiment of thedatabase 810. The database 810 includes, for example, an ar contentdataset 902, a user dataset 906, a location dataset 908, a machinedataset 910, a dedicated tool board dataset 904, and an ar task dataset912.

The ar content dataset 902 includes virtual content associated with atask. For example, the virtual content may include an animation ofvirtual objects illustrating how to change a filter of an engine. Theuser dataset 906 may include identification data related to the user ofthe corresponding head mounted device. The location dataset 908 includesdata related to a geographical location of the head mounted device, ageographical location of a machine, and a geographical location of atool. The machine dataset 910 includes data related to the machine(e.g., location, function, operation, make, model, type, image). Thededicated tool board dataset 904 includes data related to the dedicatedtool board 404. For example, the data may include the number of toolsconfigured to be stored on the dedicated tool board 404, anidentification of which tools are present and absent on the dedicatedtool board 404, a location of the tools missing from the dedicated toolboard 404. The ar task dataset 912 includes data related to the tasks.For example, each task may be associated with a corresponding set ofspecific tools.

FIG. 10 is a table illustrating an example of a table 1020 of a dataset.The table 1020 includes fields for user 1002, location 1004, machine1006, task 1008, tools for task 1010, detected tool(s) 1012, and arcontent 1018. The table 1020 illustrates example entries 1014 and 1016.For example, entries 1014 illustrate an example of a user, John, islocated at building x. An engine A is within a field of view of the headmounted device worn by John. John's task as identified in the augmentedreality application is to replace a filter. The task requires a specificwrench type B. The head mounted device of John detects that John has thewrench type C (wrench type C is located within a field of view of thehead mounted device). At AR content provided to the head mounted deviceincludes a virtual wrench type B. Thus, John may perceive a virtualwrench type B hovering above the engine A to indicate the right type oftool needed to replace the filter of engine A. A visual indicator may bedisplayed on top of the wrench type C to tell John that the wrench typeC is not the correct wrench for the task of replacing the filter ofengine A. For example, the visual indicator appears as a virtual crossflashing on top of the wrench type C.

Entries 1016 illustrate an example of a user, Jane located at the 2ndfloor of a builder. The head mounted device of Jane detects that theengine B is within a field of view of Jane or Jane is located next tothe engine B. The task of Jane as identified in the augmented realityapplication is to top off fluids with fluid D. The head mounted deviceof Jane does not detect any fluid around Jane and displays an image or avirtual 3D model of the fluid D for the task of topping off fluids.

FIG. 11 is an interaction diagram illustrating an example ofinteractions between head mounted devices and the server 402. Atoperation 1102, the head mounted device 406 provides a dataset to theserver 402. The dataset includes, for example, sensor data from the headmounted device 406 that identifies a geographic location of the headmounted device 406, the task selected or assigned to the augmentedreality application in the head mounted device 406, physical objectsdetected within a field of view of the head mounted device 406, and theuser profile. At operation 1104, the head mounted device 412 provides adataset related to the machine 420 and the tool 426 visually presentwithin a field of view of the head mounted device 412 to the server 402.At operation 1108, the head mounted device 416 provides a datasetrelated to the dedicated tool board 404 to the server 402. For example,the dataset from head mounted device 416 includes an identification ofwhich tools are present and missing from the dedicated tool board 404.

At operation 1110, the server 402 generates an inventory of tools basedon the datasets received from head mounted device 406, 412, 416. Theinventory of tools may identify the location of each tool within thelocation 410 and the corresponding head mounted device user.

At operation 1112, the server 402 performs a tool compliance audit foreach head mounted device based on the inventory of the tools and thedataset for each head mounted device. The tool compliance audit verifiesthat the tools are located with corresponding task at the head mounteddevices.

At operation 1114, the server 402 generates and sends ar content datasetcorresponding to non-compliance of the head mounted device 406. Forexample, the ar content dataset warns the user 414 of the head mounteddevice 406 that the tools for the task are not detected and present atthe head mounted device 406. The ar content dataset may include visualindicators and 3D models of virtual objects to be rendered in the headmounted device 406.

At operation 1116, the server 402 generates and sends ar content datasetcorresponding to an incorrect tool at the head mounted device 412. Forexample, the ar content dataset warns the user 422 of the head mounteddevice 412 that the detected tool at the head mounted device 412 is notthe correct tool for the task. The ar content dataset may include visualindicators and 3D models of virtual objects to be rendered in the headmounted device 412.

At operation 1118, the server 402 generates and sends ar content datasetcorresponding to a tool being absent or missing in the dedicated toolboard 1614. For example, the ar content dataset includes displayingwhich user has the tool on the dedicated tool board 404. The ar contentdataset may include visual indicators and 3D models of virtual objectsto be rendered in the head mounted device 416 pointed at the dedicatedtool board 404.

FIG. 12 is a diagram illustrating an example of virtual contentdisplayed in a transparent display of a head mounted device.

The transparent display 1202 is aimed at machine 1208. The transparentdisplay 1202 displays virtual content showing tools needed foridentified task 1204 (e.g., virtual 3D model of a hammer) and virtualcontent identifying task in ar application 1206 (e.g., displaying atitle or description of the task).

FIG. 13 is a diagram illustrating another example of virtual contentdisplayed in a transparent display of a head mounted device.

The transparent display 1302 is aimed at the machine 1308 and the tool1310. The transparent display 1302 displays a layer of virtual contentthat includes virtual content showing validation of tool for task 1304(e.g., green glowing outline or bubble around the tool 1310, a checkmarkon the tool 1310) and virtual content identifying task in ar application1306.

FIG. 14 is a diagram illustrating another example of virtual contentdisplayed in a transparent display of a head mounted device.

The transparent display 1408 is aimed at the machine 1404 and the tool1406. The transparent display 1408 displays a layer of virtual contentthat includes virtual content showing validation of tool for task 1410,virtual content identifying task in ar application 1412, and virtualcontent showing additional tool for task 1402 (e.g., 3D model of ascrewdriver displayed next to the tool 1406).

FIG. 15 is a diagram illustrating another example of virtual contentdisplayed in a transparent display of a head mounted device.

The transparent display 1502 is aimed at the machine 112 and the tool128. The transparent display 1502 displays a layer of virtual contentthat includes virtual content highlighting incorrect tool for task 1504(e.g., virtual red cross displayed on top of the tool 128), virtualcontent identifying task in ar application 1306, and virtual contentshowing correct tool for task 1506.

FIG. 16 is a diagram illustrating an example of virtual contentdisplayed in a transparent display of a head mounted device pointed at adedicated tool board.

A transparent display 1602 may be pointed at the dedicated tool board1614. Tools 502, 514, 504, 510 are present on the dedicated tool board1614. A layer of virtual objects is displayed in the transparent display1602. The virtual objects include virtual object 1604, virtual object1612, virtual object 1610, virtual arrow for correct tool 1608, andvirtual object 1606.

The virtual objects 1604, 1610, 1612 are displayed at the predefinedlocations of the respective tools. For example, virtual object 1604 mayinclude a picture of the user displayed on the location assigned to thetool detected at the head mounted device of the corresponding user.Similarly, virtual object 1610 is displayed at the location on thededicated tool board 1614 where the corresponding tool is missing.

The virtual object 1612 is displayed on a location assigned to a missingtool. The server 402 detects that the missing tool at the correspondinglocation on the dedicated tool board 1614 is an incorrect tool andgenerates a virtual arrow for correct tool 1608 linking the correct tool(e.g., tool 504) to the virtual object 1612 (e.g., picture of the user).

The virtual object 1606 is displayed on top of the tool 510 to identifythat the corresponding user does not have the proper tool (tool 510) toperform a task.

FIG. 17 is a flowchart illustrating an example operation of generating areal time inventory of tools at a mobile device.

At block 1702, the mobile device 104 receives or downloads a datasetfrom the server 124. In one example embodiment, block 1702 may beimplemented with the tool inventory module 212.

At block 1704, the mobile device 104 (passively) identifies and locatestools within the location 102. In one example embodiment, block 1704 maybe implemented with the tool recognition module 202 and tool inventorymodule 212.

At block 1706, the mobile device 104 generates a tool inventory at thelocation 102 based on data from other mobile devices at the samelocation 102.

In one example embodiment, block 1704 may be implemented with the toolrecognition module 202 and tool inventory module 212.

FIG. 18 is a flowchart illustrating an example operation of generatingan augmented reality content dataset at a head mounted device.

At block 1802, the head mounted device 406 receives or downloads adataset from the server 402. In one example embodiment, block 1902 maybe implemented with the tool mapping and localization module 612. Inanother example, the server 124 may push the ar task dataset 912 to thehead mounted device 406.

At block 1804, the head mounted device 406 identifies and locates toolsbased on the tools specified in a selected ar task (e.g., user-selectedor pre-assigned task). In one example embodiment, block 1804 may beimplemented with the tool mapping and localization module 612.

At block 1806, the head mounted device 406 identifies when new tools arefound on the tool board or tools are placed back on the tool board. Inone example embodiment, block 1806 may be implemented with the toolmapping and localization module 612.

At block 1808, the head mounted device 406 checks a running toolinventory with any new ar tasks. In other words, the head mounted device406 determines which tools are required for a particular ar task. In oneexample embodiment, block 1808 may be implemented with the head mounteddevice 406.

FIG. 19 is a flowchart illustrating an example operation 1900 ofgenerating an augmented reality content dataset for a head mounteddevice at the server 402.

At block 1902, the server 402 receives dataset or sensor data from thehead mounted devices. In one example embodiment, block 1902 may beimplemented with external sensor interface module 802 and head mounteddisplay interface module 804.

At block 1904, the server 402 generates an inventory of the tools basedon the datasets received at block 1902. In one example embodiment, block1904 may be implemented with the server inventory application 806.

At block 1906, the server 402 identifies incorrect and missing tools atthe respective head mounted device. In one example embodiment, block1906 may be implemented with the server compliance application 808.

At block 1908, the server 402 generates an ar dataset for each headmounted device based on the identified incorrect or missing tools. Inone example embodiment, block 1908 may be implemented with the servercompliance application 808.

At block 1910, the ar dataset is sent to the corresponding head mounteddevice. In one example embodiment, block 1902 may be implemented withserver compliance application 808.

FIG. 20 is a flowchart illustrating an example operation of generatingan augmented reality content dataset related to non-compliance for ahead mounted device.

At block 2002, the server 402 identifies present and absent tools on thededicated tool board 1614. In one example embodiment, block 2002 may beimplemented with the external sensor interface module 802 and the headmounted display interface module 804.

At block 2004, the server 402 compares the present and absent tools witha tool associated with an ar task of a head mounted device. In oneexample embodiment, block 2004 may be implemented with the servercompliance application 808 and the server inventory application 806.

At block 2006, the server 402 identifies a non-compliance if a toolassociated with an ar task is still present on the dedicated tool board1614. In one example embodiment, block 2006 may be implemented with theserver compliance application 808.

At block 2008, the server 402 generates an ar dataset representing thenon-compliance and sends the ar dataset to the head mounted device. Inone example embodiment, block 2008 may be implemented with the servercompliance application 808.

FIG. 21 is a flowchart illustrating an example operation of generatingan augmented reality content dataset related to an incorrect tool for ahead mounted device at a server.

At block 2102, the server 402 detects and identifies tools present andabsent on the dedicated tool board 1614. In one example embodiment,block 2102 may be implemented with the server inventory application 806.

At block 2104, the server 402 compares tools present on the dedicatedtool board 1614 with tools associated with an ar task of a head mounteddevice and a tool detected at the head mounted device. In one exampleembodiment, block 2104 may be implemented with the server complianceapplication 808.

At block 2106, the server 402 identifies an incorrect tool at the headmounted device based on the comparison in block 2104. In one exampleembodiment, block 2106 may be implemented with the server complianceapplication 808.

At block 2108, the server 402 generates and sends an ar dataset relatedto the incorrect tool to the head mounted device. In one exampleembodiment, block 2108 may be implemented with the server complianceapplication 808.

FIG. 22 is a flowchart illustrating an example operation 2200 ofdisplaying a visual indicator for alignment in a transparent display ofa head mounted device at a server.

At block 2202, the server 402 identifies tools absent from the dedicatedtool board 1614. In one example embodiment, block 2202 may beimplemented with the external sensor interface module 802, the headmounted display interface module 804, and the server inventoryapplication 806.

At block 2204, the server 402 identifies tools present at the dedicatedtool board 1614. In one example embodiment, block 2204 may beimplemented with the server inventory application 806.

At block 2206, the server 402 generates an ar dataset to show which headmounted device/user on the dedicated tool board 1614. In one exampleembodiment, block 2008 may be implemented with the server inventoryapplication 806.

FIG. 23 is a flowchart illustrating an example operation of generatingan augmented reality content dataset related to a dedicated tool boardat a server.

At block 2302, the server 402 identifies present and absent tools at thededicated tool board 1614. In one example embodiment, block 2302 may beimplemented with the server inventory application 806.

At block 2304, tools present at each head mounted device are identified.In one example embodiment, block 2304 may be implemented with the serverinventory application 806.

At block 2306, an ar dataset showing which tool (that is currentlypresent on the dedicated tool board 1614) should be used at the headmounted device. In one example embodiment, block 2306 may be implementedwith the server compliance application 808.

FIG. 24 is a block diagram illustrating components of a machine 2400,according to some example embodiments, able to read instructions 2406from a computer-readable medium 2418 (e.g., a non-transitorymachine-readable medium, a machine-readable storage medium, acomputer-readable storage medium, or any suitable combination thereof)and perform any one or more of the methodologies discussed herein, inwhole or in part. Specifically, the machine 2400 in the example form ofa computer system (e.g., a computer) within which the instructions 2406(e.g., software, a program, an application, an applet, an app, or otherexecutable code) for causing the machine 2400 to perform any one or moreof the methodologies discussed herein may be executed, in whole or inpart.

In alternative embodiments, the machine 2400 operates as a standalonedevice or may be communicatively coupled (e.g., networked) to othermachines. In a networked deployment, the machine 2400 may operate in thecapacity of a server machine or a client machine in a server-clientnetwork environment, or as a peer machine in a distributed (e.g.,peer-to-peer) network environment. The machine 2400 may be a servercomputer, a client computer, a personal computer (PC), a tabletcomputer, a laptop computer, a netbook, a cellular telephone, asmartphone, a set-top box (STB), a personal digital assistant (PDA), aweb appliance, a network router, a network switch, a network bridge, orany machine capable of executing the instructions 2406, sequentially orotherwise, that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute the instructions 2406 to perform all or part of any oneor more of the methodologies discussed herein.

The machine 2400 includes a processor 2404 (e.g., a central processingunit (CPU), a graphics processing unit (GPU), a digital signal processor(DSP), an application specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), or any suitable combinationthereof), a main memory 2410, and a static memory 2422, which areconfigured to communicate with each other via a bus 2412. The processor2404 contains solid-state digital microcircuits (e.g., electronic,optical, or both) that are configurable, temporarily or permanently, bysome or all of the instructions 2406 such that the processor 2404 isconfigurable to perform any one or more of the methodologies describedherein, in whole or in part. For example, a set of one or moremicrocircuits of the processor 2404 may be configurable to execute oneor more modules (e.g., software modules) described herein. In someexample embodiments, the processor 2404 is a multicore CPU (e.g., adual-core CPU, a quad-core CPU, or a 128-core CPU) within which each ofmultiple cores behaves as a separate processor that is able to performany one or more of the methodologies discussed herein, in whole or inpart. Although the beneficial effects described herein may be providedby the machine 2400 with at least the processor 2404, these samebeneficial effects may be provided by a different kind of machine thatcontains no processors (e.g., a purely mechanical system, a purelyhydraulic system, or a hybrid mechanical-hydraulic system), if such aprocessor-less machine is configured to perform one or more of themethodologies described herein.

The machine 2400 may further include a video display 2408 (e.g., aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, a cathode ray tube (CRT), orany other display capable of displaying graphics or video). The machine2400 may also include an alphanumeric input device 2414 (e.g., akeyboard or keypad), a cursor control device 2416 (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, an eye trackingdevice, or other pointing instrument), a drive unit 2402, a signalgeneration device 2420 (e.g., a sound card, an amplifier, a speaker, aheadphone jack, or any suitable combination thereof), and a networkinterface device 2424.

The drive unit 2402 (e.g., a data storage device) includes thecomputer-readable medium 2418 (e.g., a tangible and non-transitorymachine-readable storage medium) on which are stored the instructions2406 embodying any one or more of the methodologies or functionsdescribed herein. The instructions 2406 may also reside, completely orat least partially, within the main memory 2410, within the processor2404 (e.g., within the processor's cache memory), or both, before orduring execution thereof by the machine 2400. Accordingly, the mainmemory 2410 and the processor 2404 may be considered machine-readablemedia (e.g., tangible and non-transitory machine-readable media). Theinstructions 2406 may be transmitted or received over a computer networkvia the network interface device 2424. For example, the networkinterface device 2424 may communicate the instructions 2406 using anyone or more transfer protocols (e.g., hypertext transfer protocol(HTTP)).

In some example embodiments, the machine 2400 may be a portablecomputing device (e.g., a smart phone, tablet computer, or a wearabledevice), and have one or more additional input components (e.g., sensorsor gauges). Examples of such input components include an image inputcomponent (e.g., one or more cameras), an audio input component (e.g.,one or more microphones), a direction input component (e.g., a compass),a location input component (e.g., a global positioning system (GPS)receiver), an orientation component (e.g., a gyroscope), a motiondetection component (e.g., one or more accelerometers), an altitudedetection component (e.g., an altimeter), a biometric input component(e.g., a heartrate detector or a blood pressure detector), and a gasdetection component (e.g., a gas sensor). Input data gathered by any oneor more of these input components may be accessible and available foruse by any of the modules described herein.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While thecomputer-readable medium 2418 is shown in an example embodiment to be asingle medium, the term “machine-readable medium” should be taken toinclude a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storeinstructions. The term “machine-readable medium” shall also be taken toinclude any medium, or combination of multiple media, that is capable ofstoring the instructions 2406 for execution by the machine 2400, suchthat the instructions 2406, when executed by one or more processors ofthe machine 2400 (e.g., processor 2404), cause the machine 2400 toperform any one or more of the methodologies described herein, in wholeor in part. Accordingly, a “machine-readable medium” refers to a singlestorage apparatus or device, as well as cloud-based storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-readable medium” shall accordingly be taken to include,but not be limited to, one or more tangible and non-transitory datarepositories (e.g., data volumes) in the example form of a solid-statememory chip, an optical disc, a magnetic disc, or any suitablecombination thereof. A “non-transitory” machine-readable medium, as usedherein, specifically does not include propagating signals per se. Insome example embodiments, the instructions 2406 for execution by themachine 2400 may be communicated by a carrier medium. Examples of such acarrier medium include a storage medium (e.g., a non-transitorymachine-readable storage medium, such as a solid-state memory, beingphysically moved from one place to another place) and a transient medium(e.g., a propagating signal that communicates the instructions 2406).

What is claimed is:
 1. A method comprising: receiving, at a server,sensor data from a plurality of mobile devices having optical sensorsand location sensors, the sensor data identifying a mobile physicalobject and a location of the mobile physical object within a predefinedgeographic region based on the optical and location sensors; identifyingmobile physical objects that are present and absent within thepredefined geographic region based on a predefined catalog of the mobilephysical objects at the predefined geographic region; generating, at theserver, a real-time inventory of the mobile physical objects at thepredefined geographic region based on the mobile physical objects thatare present and absent within the predefined geographic region, thereal-time inventory including the identification of the mobile physicalobjects and the location of the mobile physical objects within thegeographic region; receiving datasets from the plurality of mobiledevices, each dataset identifying a task selected in an augmentedreality application of a corresponding mobile device, an identificationof a physical tool detected at the corresponding mobile device, eachmobile device including a display device; identifying physical toolspresent and absent at a dedicated tool board, the dedicated tool boardconfigured to store the physical tools for the tasks of the augmentedreality application; comparing an identification of the physical toolspresent and absent at the dedicated tool board with the physical toolsdetected at the mobile devices and the tasks identified at the mobiledevices to generate a tool inventory and a tool compliance, the toolinventory identifying physical tools absent from the dedicated toolboard and detected at the corresponding mobile device, the toolcompliance identifying whether the physical tool detected at thecorresponding mobile device is valid for the task selected in theaugmented reality application of the corresponding mobile device;generating an augmented reality content dataset for each mobile device,each augmented reality content dataset comprising a virtual objectidentifying at least one of a missing physical tool, an incorrectphysical tool, and a valid physical tool based on the tool compliance;and generating a dedicated tool board augmented reality content datasetfor the dedicated tool board based on the tool inventory, the dedicatedtool board augmented reality content dataset comprising a plurality ofvirtual objects identifying users of the mobile devices withcorresponding physical tools on the dedicated tool board.
 2. The methodof claim 1, wherein the dataset further comprises an identification of auser for each mobile device, an identification and a location of thephysical object within a field of view of each mobile device, the taskidentifying a physical operation to perform on the physical object, theaugmented reality application configured to generate virtual objectsdisplayed in a transparent display of the corresponding mobile device,the virtual objects including a visual illustration of how to performthe task and how operate the physical tool related to the task on thephysical object, wherein the method further comprises: identifyingdataset relevant to a corresponding mobile device and updating a localrecognition content dataset at the corresponding mobile device with therelevant dataset.
 3. The method of claim 1, wherein the dedicated toolboard includes a plurality of tool identifiers, the plurality of toolidentifiers including a plurality of outlines displayed on the dedicatedtool board, each outline corresponding to a physical tool on thededicated tool board, each tool identifier generated for computer visionidentification.
 4. The method of claim 1, further comprising: using adepth sensor of a mobile device with the dedicated tool board within afield of view of the mobile device, to determine the physical toolspresent at the dedicated tool board; and identifying physical toolsabsent from the dedicated tool board based depth sensor data, a shape ofan outline, and a location of the outline relative to the dedicated toolboard.
 5. The method of claim 1, wherein the virtual object includes athree-dimensional model of a physical tool related to the task selectedat the corresponding mobile device.
 6. The method of claim 1, whereinthe virtual object includes a first visual indicator to validate a firstphysical tool detected at the corresponding mobile device for the taskselected at the corresponding mobile device.
 7. The method of claim 1,wherein the virtual object includes a second visual indicator toidentify the second physical tool detected at the corresponding mobiledevice as an incorrect physical tool for the task selected at thecorresponding mobile device.
 8. The method of claim 1, wherein thededicated tool board augmented reality content dataset comprises atleast one of: a first virtual object identifying a user of a mobiledevice corresponding to a physical tool absent from the dedicated toolboard; a second virtual object comprising a visual indicator linking aphysical tool present on the dedicated tool board to the first virtualobject; a third virtual object identifying a user of a mobile devicecorresponding to a physical tool present on the dedicated tool board;and a fourth virtual object comprising a visual indicator of a currentlocation of a physical tool absent from the dedicated tool board.
 9. Themethod of claim 1, further comprising: communicating the augmentedreality content dataset to the corresponding mobile device; causing adisplay of the augmented reality content dataset in a transparentdisplay of the corresponding mobile device; communicating the dedicatedtool board augmented reality content dataset to a mobile device with thededicated tool board within a field of view of the mobile device; andcausing the display of the dedicated tool board augmented realitycontent dataset in the transparent display of the mobile device with thededicated tool board within the field of view of the mobile device. 10.A server comprising: a processor; and a memory storing instructionsthat, when executed by the processor, configure the server to: receivesensor data from a plurality of mobile devices having optical sensorsand location sensors, the sensor data identifying a mobile physicalobject and a location of the mobile physical object within a predefinedgeographic region based on the optical and location sensors; identifymobile physical objects that are present and absent within thepredefined geographic region based on a predefined catalog of the mobilephysical objects at the predefined geographic region; generate areal-time inventory of the mobile physical objects at the predefinedpredefined geographic region based on the mobile physical objects thatare present and absent within the predefined geographic region, thereal-time inventory including the identification of the mobile physicalobjects and the location of the mobile physical objects within thepredefined predefined geographic region; receive datasets from theplurality of mobile devices, each dataset identifying a task selected inan augmented reality application of a corresponding mobile device, anidentification of a physical tool detected at the corresponding mobiledevice, each mobile device including a display device; identify physicaltools present and absent at a dedicated tool board, the dedicated toolboard configured to store the physical tools for the tasks of theaugmented reality application; compare an identification of the physicaltools present and absent at the dedicated tool board with the physicaltools detected at the mobile devices and the tasks identified at themobile devices to generate a tool inventory and a tool compliance, thetool inventory identifying physical tools absent from the dedicated toolboard and detected at the corresponding mobile device, the toolcompliance identifying whether the physical tool detected at thecorresponding mobile device is valid for the task selected in theaugmented reality application of the corresponding mobile device;generate an augmented reality content dataset for each mobile device,each augmented reality content dataset comprising a virtual objectidentifying at least one of a missing physical tool, an incorrectphysical tool, and a valid physical tool based on the tool compliance;and generate a dedicated tool board augmented reality content datasetfor the dedicated tool board based on the tool inventory, the dedicatedtool board augmented reality content dataset comprising a plurality ofvirtual objects identifying users of the mobile devices withcorresponding physical tools on the dedicated tool board.
 11. The serverof claim 10, wherein the dataset further comprises an identification ofa user for each mobile device, an identification and a location of thephysical object within a field of view of each mobile device, the taskidentify a physical operation to perform on the physical object, theaugmented reality application configured to generate virtual objectsdisplayed in a transparent display of the corresponding mobile device,the virtual objects include a visual illustration of how to perform thetask and how operate the physical tool related to the task on thephysical object.
 12. The server of claim 10, wherein the dedicated toolboard includes a plurality of tool identifiers, the plurality of toolidentifiers include a plurality of outlines displayed on the dedicatedtool board, each outline corresponding to a physical tool on thededicated tool board, each tool identifier generated for computer visionidentification.
 13. The server of claim 10, wherein the instructionsfurther configure the server to: using a depth sensor of a mobile devicewith the dedicated tool board within a field of view of the mobiledevice, to determine the physical tools present at the dedicated toolboard; and identify physical tools absent from the dedicated tool boardbased depth sensor data, a shape of an outline, and a location of theoutline relative to the dedicated tool board.
 14. The server of claim10, wherein the virtual object includes a three-dimensional model of aphysical tool related to the task selected at the corresponding mobiledevice.
 15. The server of claim 10, wherein the virtual object includesa first visual indicator to validate a first physical tool detected atthe corresponding mobile device for the task selected at thecorresponding mobile device, and a second a visual indicator to identifya second physical tool detected at the corresponding mobile device as anincorrect physical tool for the task selected at the correspondingmobile device.
 16. The server of claim 10, wherein the dedicated toolboard augmented reality content dataset comprises at least one of: afirst virtual object identify a user of a mobile device corresponding toa physical tool absent from the dedicated tool board; a second virtualobject comprising a visual indicator link a physical tool present on thededicated tool board to the first virtual object; a third virtual objectidentify a user of a mobile device corresponding to a physical toolpresent on the dedicated tool board; and a fourth virtual objectcomprising a visual indicator of a current location of a physical toolabsent from the dedicated tool board.
 17. The server of claim 10,wherein the instructions further configure the server to: communicatethe augmented reality content dataset to the corresponding mobiledevice; cause a display of the augmented reality content dataset in atransparent display of the corresponding mobile device; communicate thededicated tool board augmented reality content dataset to a mobiledevice with the dedicated tool board within a field of view of themobile device; and cause the display of the dedicated tool boardaugmented reality content dataset in the transparent display of themobile device with the dedicated tool board within the field of view ofthe mobile device.
 18. A non-transitory computer-readable storagemedium, the computer-readable storage medium including instructions thatwhen executed by a computer, cause the computer to: receive sensor datafrom a plurality of mobile devices having optical sensors and locationsensors, the sensor data identifying a mobile physical object and alocation of the mobile physical object within a predefined geographicregion based on the optical and location sensors; identify mobilephysical objects that are present and absent within the predefinedgeographic region based on a predefined catalog of the mobile physicalobjects at the predefined geographic region; generate a real-timeinventory of the mobile physical objects at the predefined geographicregion based on the mobile physical objects that are present and absentwithin the predefined geographic region, the real-time inventoryincluding the identification of the mobile physical objects and thelocation of the mobile physical objects within the predefined geographicregion; receive datasets from the plurality of mobile devices, eachdataset identifying a task selected in an augmented reality applicationof a corresponding mobile device, an identification of a physical tooldetected at the corresponding mobile device, each mobile deviceincluding a display device; identify physical tools present and absentat a dedicated tool board, the dedicated tool board configured to storethe physical tools for the tasks of the augmented reality application;compare an identification of the physical tools present and absent atthe dedicated tool board with the physical tools detected at the mobiledevices and the tasks identified at the mobile devices to generate atool inventory and a tool compliance, the tool inventory identifyingphysical tools absent from the dedicated tool board and detected at thecorresponding mobile device, the tool compliance identifying whether thephysical tool detected at the corresponding mobile device is valid forthe task selected in the augmented reality application of thecorresponding mobile device; generate an augmented reality contentdataset for each mobile device, each augmented reality content datasetcomprising a virtual object identifying at least one of a missingphysical tool, an incorrect physical tool, and a valid physical toolbased on the tool compliance; and generate a dedicated tool boardaugmented reality content dataset for the dedicated tool board based onthe tool inventory, the dedicated tool board augmented reality contentdataset comprising a plurality of virtual objects identifying users ofthe mobile devices with corresponding physical tools on the dedicatedtool board.